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SYSTEM AND METHOD TO ENABLE BLIND PEOPLE TO HAVE 
ACCESS TO INFORMATION PRINTED ON A PHYSICAL DOCUMENT 

Field of the invention 

The present invention generally relates to a system and method for enabling 
5 blind and visually impaired people to interact with a physical document and to have 
access to information printed on a physical document. 

Bacl(ground of the invention 

Accessibility 

In broad terms, accessibility is the ability for a user to access and use 

10 information and technology even if this user operates in constrained conditions. 
Information and technology can be accessible through appropriate specialized technical 
interfaces. The present invention Is directed to a technology to enable visually impaired 
and blind people to access information printed on physical documents. 

Many groups, in governments, private industries and universities, are working on 

15 ways to guarantee universal access to the on-line world for all individuals in society. 
This social movement, combined with a strong push to provide a better accessibility in 
the workplace, drives an urgent call for new technologies. 

As information technology (IT) penetrates all commercial and public transactions 
and communications, it is important to ensure accessibility to everyone. Governments 

20 have begun to define new regulations and standards to enable people with disabilities 
to easily access information technology. For instance, in the US, the federal, state and 
local governments have initiated a legislation requiring computer technology to be 
accessible to people with disabilities. More particularly, in June 2001, the U.S. Federal 
Government has implemented a legislation known as "Section 508". Section 508 has 

25 changed the rules of the game on how the government must buy information 
technology and in particular "accessible" IT. 
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Countries have established regulations that require accessibility at some level, 
other countries are in the process of developing such regulations. In addition, 
international organizations are working to develop accessibility standards. 

Accessibility Aids for Blind People 

5 For someone who is completely unable to use a normal screen or read a printed 

text, there are two alternatives: Braille reading or speech. Currently available assistance 
for blind and visually impaired people comprises a wide range of technical solutions, 
including document scanners and enlargers, interactive speech software and cognitive 
tools, screen reader software and screen enlargement programs. 

10 • Screen Readers: This technology converts text (or other elements) on a computer 
screen to speech, allowing blind or visually impaired users to hear what is displayed 
on their computer screen. For example, "Jaws for Window^' is a computer 
screen-reader application from Henter- Joyce. It operates with Windows NT 4.0, 
Windows 98 and Windows 95 and includes an enhanced software speech 

15 synthesizer for speaking in several languages. 

Note : Windows is a trademark of Microsoft Corporation, a company with corporate 
headquarters in Redmond, Washington, USA. 

• Web Page Readers: This technology reads aloud text just as it appears on Web 
pages. For example, "Home Page Reader" of IBM, understands HTML (Hypertext 
20 Markup Language), the programming language used to design Web pages. Home 

Page Reader can accurately read text comprised in tables, or included in graphic 
descriptions, so that blind users can easily navigate on the Web. 
Note : IBI\/I is a trademark of International Business Machines Corporation, a 
company with corporate headquarters in Armonk, New York, USA. 

25 • Mall Readers: This technology uses a screen reader for converting the text of an 
e-mail to speech. For example, "Lotus Notes Release 5" of IBM, enables blind users 
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to check the status of their e-mail (unread, read, deleted), to use calendar functions, 
and to file and organize documents. 

• Braille Editing Systems (BES): These systems allow users to input Braille 
characters from the keyboard. For instance, IBM provides a BES that can be 
downloaded from the IBM Web. 

• Scanning Reading Systems: The information printed on a physical document is 
first scanned. Then, using an OCR (Optical Character Recognition) technique and a 
speech synthesizer, this information can be read to blind people. For example, 
"Open Book: Ruby Edition 4.0" oi IBM is a program that transforms a computer, a 
sound card, and a scanner into a reading machine for reading books, magazines, 
memos, bills, and many other printed documents. 

• Screen l\/lagnifiers: These software tools allow a visually impaired user to magnify 
text and images on a computer screen. For example, "MAGic Screen Magnification", 
from Henter-Joyce, is a screen magnification product for Windows. 

• Portable document magnifiers: The MAXPort by NanoPac, Inc. is a portable 
magnifying system. The user wears a special pair of glasses, points at a particular 
part of a physical document and can see a magnified image of said particular part. 

Access to Printed l\/laterial 

Reading manuals and reports at work, textbooks at school, and menus at 
restaurants, and more generally reading printed material at any time and in any place is 
part of daily life. The present invention is based on the following obsen/ation: it is easier 
and quicker for sighted people to turn over pages of a book than to browse electronic 
pages on a computer screen. It would be an improvement to provide a similar capability 
to blind and visually impaired people. For instance, it would be a real improvement for 
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blind people to have the possibility to access information printed on a physical 
document directly from this physical document. 

When we compare paper based information with computer based information, 
paper has a number of useful properties that computers cannot provide. For instance: 

5 • paper is portable, familiar and can be easily distributed; 

• paper is easy to read, mark, and manipulate. 

Many electronic systems attempt to replace paper. They offer, for instance, a 
better access to multimedia services. However, most users prefer to work with paper 
and it is difficult to foresee, in a near future, a general and massive replacement of 
10 paper books by electronic books (e.g., by Web accessible e-books). 

Publication entitled "The Last BooK\ IBM Systems Journal, Vol 36, No. 3 - 1997, 
by J. Jacobson, B. Comiskey, C. Turner, J. Albert, and P. Tsao of the MIT Media 
Laboratory, compares printed books and computer screens in the following terms: 

"A book represents a fundamentally different entity than a computer screen in 
15 that it is a physical embodiment of a large number of simultaneous high-resolution 

displays. When we turn the page, we do not lose the previous page. Through 
evolution the brain has developed a highly sophisticated spatial map. Persons 
familiar with a manual or textbook can find information that they are seeking with 
high specificity, as evidenced by their ability to remember whether something that 
20 was seen only briefly was on the right side or left side of a page, for instance. 

Furthermore their haptic connection with the brain's spatial map comprises a highly 
natural and effective interface, when such information is embodied on actual 
multiple physical pages. 

Another aspect of embodying information on multiple, simultaneous pages is that 
25 of serendipity and comparison. We may leaf through a large volume of text and 
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graphics, inserting a finger bool<marl< into tliose areas of greatest interest Similarly, 
we may assemble a large body of similar matter in order to view elements in 
contrast to one another, such as might be done to determine which of a particular 
set of graphical designs is most satisfying". 

5 Publication entitled "A Comparison of Reading Paper and On-Line Document^', 

O'Hara and Sellen, XRCE: Technical Report: EPC-1 997-1 01, Xerox Co. summarizes a 
study comparing reading from paper to reading on-line. As reported in this article : 

"Critical differences have to do with the major advantages paper offers in supporting 
annotation while reading, quick navigation, and flexibility of spatial layout These, in 
10 turn, allow readers to deepen their understanding of the text, extract a sense of its 

structure, create a plan for writing, cross-refer to other documents, and interleave 
reading and writing". 

As reported in an article entitled ''Paper is still with usl\ by Ph. Jane Benson - 
The Journal of Electronic Publishing, published by the University of Michigan Press - 

15 "Studies of professionals at work underscore three points that are critical to the 

design of scientific information: paper isn't going away, reading and writing are 
inextricably intertwined, and readers sample and navigate text according to specific 
purposes and tasks". 

Also, in the article entitled "Electronic Journals : What do users think of them ?\ 
20 by Dr Cliff McKnight, Department of Information and Library Studies, Loughborough 
University, Loughborough, Leicester LE1 1 3TU, UK, we can read : 

"A paper journal article is more easily manipulated than an electronic version. As 
one student subject in the Cafe Jus study remarked: It is easier to turn the page of a 
paper journal'. More than just page-turning, though, regular journal readers 
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understand the structure of journal articles and use that knowledge to aid 
manipulation, providing rapid access to the information in an article. For example, 
Dillon showed that people could reconstruct articles from a set of paragraphs with 
about 80% accuracy. They also displayed a similar level of accuracy in saying which 
5 section of an article isolated sentences had been taken from. Watching a reader 

pick up an issue of a journal, skim through the contents, turn to an article and skim 
through it and then tum back to the contents, all in a matter of seconds, it is clear 
that such skills are largely automatic. Watching people trying to do the same thing in 
an electronic journal system, it is clear that people will have to learn a different set of 
10 skills". 

Sighted people can browse very easily through paper catalogs, magazines, 
newspapers, maps and books by flipping through the pages and by "glancing" at 
pictures and text. It is also very easy for them to mark and return to specific parts of a 
physical document. It would be desirable to offer similar possibilities to blind and 
15 visually impaired people and in particular to enable these people to easily browse 
through the same paper catalogs, magazines, newspapers, maps and books, and to 
easily "access" (receive and hear) information related to pictures and text printed on 
pages of these documents. 

The convenience and usefulness of a system for enabling blind and visually 
20 impaired people to access information directly from hard-copy documents, to turn over 
pages and to manually search for text, is well illustrated by in the following article 
entitled "Talking Books Speak Volumes", by Kendra Mayfield, Wired News, July 13, 
2000, 

"When George Kerscher began to lose his sight in his late twenties, the former 
25 teacher realized that he might never be able to read the same way again. Since he 

could no longer delve through the written page Kerscher, who has the degenerative 
disease retinitis pigmentosa, began listening to audio books. But while these 
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cassettes helped him read, they were limited. He couldn't turn pages or search for 
text. He had to endure hours of recordings to find the information he needed". 

In fact audio tapes have a drawback which is similar to the drawback 
experienced when we scroll connputer screens: the user can only go forward or 
5 backwards. The possibilities of navigation are very limited. It is difficult for people with 
visual impairments, to use audio books or screen readers the same way sighted people 
do. For instance, looking up a recipe or searching for a gardening tip on an audio tape 
or on a long electronic book is extremely tedious and time consuming. 

Nowadays, digital audio books are designed to make published documents 
10 accessible and navigable for to blind or visually impaired persons. The ANSI/NISO 
Z39.86 standard, defines a format and content for an electronic file set that comprises a 
digital audio book and establishes a limited set of requirements for digital audio books 
playback devices. It uses established and new specifications to delineate the structure 
of digital audio books whose content can range from text with corresponding spoken 
15 audio, to audio with little or no text. However, if a sighted user of a physical book wishes 
to discuss a particular section of such this book with a blind user, the blind user must 
actively search for a digital copy of the same edition of the book and then must search 
for the relevant section. There is a need to enable individuals familiar with paper 
documents to continue to use those same documents as they age and their vision may 
20 be deteriorating. There is also a need to provide a mechanism by which the text of a 
familiar physical document may be supplemented by speech output or display of a 
more easily readable version of the text. 

One of the characteristics of the publishing industry is that most publications can 
be delivered in a digital form. In some cases, the digital form has really supplanted the 
25 physical one. In other cases, the digital form appears complementary to the physical 
one. However, in most cases, the physical form is till dominant. In fact, in terms of 
business models, with the exception of journals and directories, which tend to follow a 
subscription model, most publishers have been unable to create profitable digital 
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products and services, able to generate significant incomes. Digital audio books 
represent an important improvement compared to traditional analog audio books. For 
instance, digital audio books provide access to virtually stored information, offer 
improved navigation capabilities, allow the use of keywords for searching and directly 
accessing text in electronic books. However, due to a relatively limited number of vision 
impaired users, the cost of such digital audio books remains relatively high. It results 
from this that a very small number of titles are published both, as printed books and as 
digital audio books. 

The necessity of using a natural friendly interface for having access to 
information has been precisely summarized by Ann Light, in the article entitled 
"Fourteen Users in Search of a Newspaper: the Effect of Expectation on Online 
Behaviour"', School of Cognitive and Computing Sciences, University of Sussex, 
CSRP507, 

"People expect the friendly familiar paradigm of media to guide them through uncharted 
territories of information". 

According to prior art, electronic reading machines using computer-based optical, 
character recognition (OCR) are widely used to help visually impaired people and 
people with reading difficulties to read information on paper documents. An electronic 
reading machine basically comprises 

• a personal computer connected to a scanner and operating with an OCR software, 
and 

• text-to-speech hardware and software. 

Usually, a method for electronically reading a printed document comprises the 
following steps : 

• placing a page of a text document on a scanner; 

• digitizing this page for obtaining a pixel bit map of the page; 
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• converting this image to text by means of an OCR program executed in a personal 
computer, this personal computer being connected to the scanner; and 

• generating speech from the text by means of a text-to-speech program. 

Currently, reading machines are provided by a variety of companies, including 
IBM, Telesensory, Arkenstone and Kurzweil Educational Systems. Unfortunately, 
electronic reading machines suffer from a variety of functional and operational 
deficiencies that limit their usefulness. For instance: 

• scanning a page is a mechanical operation. It takes time to move the electro-optical 
components of the scanner over the page. 

• it takes time also to transmit the information from the scanner to the computer. 

• the OCR process for an entire scanned page, takes a lot of time and resources. 

• only a single page can be scanned at the same time. 

• OCR is prone to error. Prime Recognition, in an advertisement for the product 
PrimeOCR, quotes a typical error rate of 2% for conventional OCR algorithms. 
PrimeOCR itself reports an error rate of 0.4%, or 8 errors in a typical page of 2000 
characters. 

As a result, even for an experienced user, it can take more than a minute before 
an electronic reading machine can read a single scanned page. During of after the 
reading, errors must be corrected. The process is repeated for every page in the 
document. 

Since scanners are limited in size, another drawback concerning electronic 
reading machines is the reading of pages with large formats (a newspaper page for 
instance). Such pages cannot be scanned in one pass but require multiple passes. 

Moreover, a complete electronic reading system with a computer, a scanner, and 
speech hardware and software is bulky and non portable. 
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It is clear that reading a text on a paper document by means of an electronic 
reading machine involves complex manipulations and time consuming operations. 
However, there are other important deficiencies that limit in practice the acceptance 
and usefulness of electronic reading machines. Basically, the tasks required for using 
5 these electronic reading machines are far to be "natural". Reading physical documents 
by means of an electronic reading machine, has nothing in common with the automatic 
"natural" way of reading practiced by sighted people. 

A mobile scanning pen comprising an optical character recognition function, can 
also be used for producing a digital version of a printed document. Wizcom's Quicklink 
10 Pen Personal Scanner is an example of such pen. However, scanning printed words 
using such a pen can be rapidly tedious and laborious. Each line must be individually 
(and accurately) scanned and a reasonable visual acuity is required. The present 
invention does not require a good vision, as only two physical markers are required to 
define the part of the document that must be read. 

15 As a conclusion, it is important to enable blind people to easily and conveniently 

access information comprised in hard-copy publications that are read everyday by 
sighted people. 

Objects of the invention 

It is an object of the invention to enable a blind or visually impaired person, as 
20 well as a person with reading disabilities, to access information printed, on a physical 
document and information related to this printed information. 

It is another object of the invention to enable a blind or visually impaired person 
to access Information directly from a physical document without using an electronic 
reading machine based on a page scanner and/or an optical character recognition 
25 (OCR). 
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It is another object of the invention to enable a blind or visually impaired person 
to easily select a part of a physical document simply by touching (e.g., with the fingertip 
or with a pen) this physical document. 

It is a further object of the present invention to enable a blind or visually impaired 
5 person to take a physical document, to open it at any page, to select with a finger or a 
pen a part of this page and to hear the text of the selected part. 

It is also an object of the present invention to enable both magnification and 
electronic reading of a selected part of a physical document. 

It is a further object of the present invention to enable a person, after having 
10 selected a part of a printed document comprising text and images, to hear the text, to 
receive descriptions of images, and to access multimedia information or services 
related to said printed text and images. 

It is a further object of the invention to provide a system that, apart from 
providing to a blind or visually impaired person, an access to information directly from a 
15 physical document, can also be used for many other applications in the daily life, such 
as helping children to read, enabling people with reading difficulties to read, enabling 
sighted people to read in the dark, enabling people with a poor vision to read small print 
on paper documents. 

20 It is a further object of the present invention to enable individuals familiar with 

paper documents to continue to use the same documents as they age and their vision 
may be deteriorating and to supplement the text of a familiar physical document by 
speech output or display of a more easily readable version of the text. 
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Summary of the invention 



As defined in independent claims, the invention relates generally to a method, a 
system and a computer program for enabling a blind or visually impaired person, to 
access information printed on a physical document, directly from said physical 
5 document, simply by pointing and touching a desired part of said document. More 
particularly, the method comprises the steps of : 

• identifying a physical document; 

• identifying and locating an electronic copy of said identified document; 

• identifying one or a plurality of pages of said physical document and identifying a 
10 part of the identified physical document using the position of points pressed on a 

touch foil successively placed and aligned over or under said identified pages; 

• retrieving from the electronic copy of the physical document, data related to the 
selected part of the document; 

• presenting said retrieved data visually or orally on the user workstation. 

15 Further embodiments of the invention are provided in the appended dependent 

claims. 

Brief description of ttie drawings 

The novel and inventive features believed characteristic of the invention are set 
forth in the appended claims. The invention itself, however, as well as a preferred mode 
20 of use, further objects and advantages thereof, will best be understood by reference to 
the following detailed description of an illustrative detailed embodiment when read in 
conjunction with the accompanying drawings, wherein : 

• Figure 1 shows a preferred embodiment of the present invention. 
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• Figure 2 shows how, according to the present invention, a user, using a barcode 
reader, reads the address (e.g.. URL Uniform Resource Locator) of a document 
encoded on a bar code printed on this document (e.g., at the bottom right corner of 
the cover) and how information related to this document (e.g., title, author, etc.) can 

5 be retrieved from a server and read to the user by means of an electronic reading 

system. 

• Figure 3 shows how a user opens a physical document (e.g., at the page where a 
bookmark has been inserted). 

• Figure 4 shows how, according to the present invention, a user using a barcode 
10 reader, can read a page number encoded on a bar code printed at a predefined 

position on a selected page and how the system decodes and audibly informs the 
user of the selected page number. 

• Figure 5 shows how a touch foil is placed aligned over a selected page of a physical 
document according to the present invention. 

15 • Figure 6 shows how a user selects a part of the physical document by pressing a 
first point and a second point on a touch foil placed aligned over a selected page. 

• Figure 7 shows the start point and the end point of the part of the physical 
document selected by the user, said start point and end point corresponding to the 
two points over the touch foil previously pressed by the user. 

20 • Figure 8 shows how, according to the present invention, a request for retrieving 
information related to a part of a document is sent to a server. 
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• Figure 9 shows how, according to the present invention, data related to the part 
selected by the user on the physical document is extracted (in bold) from the 
electronic copy of the document stored on the Server. 

• Figure 10 shows how, according to the present invention, the text corresponding to 
the selected part of the physical document is downloaded from the server to the 
user workstation. 

• Figure 11 shows how, according with the present invention, the received text file 
can be converted to an audio file or can be magnified. 

• Figure 12 shows the steps of the method of selecting a part of a physical document 
and of providing electronic reading and screen magnification of said selected part. 

The invention itself, as well as a preferred mode of use, objects and advantages 
thereof, can be understood by reference to the following illustrative embodiment 
described by the accompanying drawings. 

Detailed description of the preferred embodiment 

The present invention addresses different problems that may arise in reading 
text, particularly problems that affect visually impaired persons or persons who cannot 
see or read correctly (e.g., in low light conditions). For instance : 

• a person driving a vehicle may have some difficulties to read a text; 

• another person may not have time to read a text; 

• bad lighting or display conditions may make difficult or impossible the reading of a 
text in a printed or displayed form. 
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The present invention allows a blind reader to receive an audible transcription of 
a selected part of a physical document (for example an audible transcription of a 
selected text in book). 

The present invention also discloses a method for selecting a part of a physical 
document simply by pressing with a fingertip or a pen a touch foil placed over said 
document; for identifying data associated with this selected part in a database 
comprising an electronic copy of the document; for extracting text from said data and 
for converting this text into speech. 

The present invention is directed to a method for use in a user system connected 
to a network, for retrieving from a server, an electronic copy of a physical document, 
and information related to said document, and for playing, displaying, reading and/or 
magnifying on the screen of this user system, the retrieved electronic copy and 
information related to said document. The method comprises the steps of: 

• identifying a physical document selected by a user, preferably by reading, by means 
of a barcode reader, a document identifier printed on said physical document at a 
predefined position; 

• identifying a first page selected by the user in said physical document, preferably by 
reading, by means of a barcode reader, a page identifier printed on said first page at 
a predefined position; 

• determining the position of a first point pressed (e.g. with a fingertip or a pen) on a 
touch foil placed and aligned over or under the identified first page, said first point 
corresponding to the start point of a part selected in said identified document by the 
user; 

if start point and end point of the selected part are on the same page : 

• determining the position of a second point pressed (e.g. with a fingertip or a pen) on 
the touch foil placed and aligned over or under the identified page of said document, 
said second point corresponding to the end point of said selected part; 
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if start point and end point are not on the same page : 

• identifying a second page selected by the user in said physical document, preferably 
by reading, by means of a barcode reader, a page identifier printed on said second 
page at a predefined position; 

5 • determining the position of a second point pressed (e.g. with a fingertip) on the 
touch foil placed and aligned over or under the identified second page of said 
document, said second point corresponding to the end point of said selected part; 

• identifying the part of the identified physical document that has been selected using 
the position (e.g. coordinates) of the start point and the end point; 

10 • identifying and locating an electronic copy of the identified document and 
information related to the selected part of the document; 

• retrieving from the electronic copy of the document, data related to the part of the 
document selected by the user; 

• extracting text data from said data and audibly reading said text data by means of a 
15 text-to-speech software; 

• optionally or alternatively, magnifying said retrieved data on the user system screen; 

• optionally or alternatively, accessing multimedia information or services related to 
said retrieved data. 

Synthetic reading and screen magnification of a selected part of a physical 
20 document 

As shown in Figure 1 , the system according to an embodiment of the present 
invention comprises: 

• a physical, i.e., hard-copy document (101) (e.g., a book) comprising one or a 
plurality of pages; 
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• a physical document interface comprising a touch foil (102) (to be placed over or 
underneath a page of this document); 

• a barcode reader (1 03); 

• a user workstation (104), connected to 

• the touch foil interface (1 02); 

• the barcode reader (1 03); 

• a communication network (105) (e.g., the Internet network); 

• a server (1 06) (e.g., a Web server). 

Physical Document 

The physical document (101) can be of any kind, for example, a newspaper, a 
legal document, a geographic map, a fiction novel, an academic text book, a technical 
book, a commercial catalog or even any other type of engraved, written, or printed 
surface. The material of the document can be paper, plastic, wood or any other 
material. For identifying the selected document (101) to the system, a bar code (107) is 
printed or attached at a well known position on the physical document (e.g., on the front 
cover, back cover or first page). In the embodiment of the invention shown in Figure 1, 
the URL (i.e., the internet address) of an electronic document is encoded in a bar code 
(107). This electronic document stored on a server (106) is the copy of a physical 
document (101). As illustrated in Figure 3, to identify pages in a document (300), (101) 
a bar code (301), (302) is printed or attached at a well known position on each 
document page (e.g., on the bottom left (301) and bottom right corners (302) of the 
open document (300)). Bar codes are read by means of a barcode reader (103) 
connected to the user workstation (104). 

In an alternate embodiment of the invention, a physical document can be 
specifically produced for the purpose of accessing information from a tactile system. 
Compared to the original paper document, this physical document can have a different 
form, to include, for instance, tactile indications (in Braille, for example) related to the 
structure and pages of the document. With such a tactile document or book, a blind 
user can quickly and independently navigate in the document, and can easily mark the 
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sections of text he wish to hear. In this embodiment, to select the same part of text, the 
sen/er determines from the points selected by the user on the physical document, the 
corresponding points in the original paper document. 

Touch Foil 

5 The touch foil (102) may be made of transparent resistive or capacitive films of 

the type used commonly to manufacture touch screens. The generated signal is 
generally proportional to the coordinates of the point pressed. One example of touch foil 
it is possible to use is the TouchTek4 (4-wire Analog Resistive Touchscreens) of 
MICRO TOUCH company (TouchTek^'^ is a trademark of the MICRO TOUCH 
10 Company). 

TouchTek4 touchscreens feature hard-coated polyester topsheets, available in 
several surface finishes. Spacer dots are available in several dot arrays, optimized for 
finger, pen and finger, or pen-only input. Electronic control is provided by a serial 
controller, or by 4-wire controllers. TouchTek4's specifications include narrow inactive 

15 border areas and compact touch sensors which allow system designers and OEMs to 
provide the largest useable screen area and full mouse emulation without sacrificing 
functionality or consuming excess power. Apart from being suited for implementing the 
functions of this invention, TouchTek4 touchscreens are commonly used in hand-held 
personal information management systems, PDAs, mobile computing systems, 

20 automotive, diagnostics and telecom devices, and Internet appliances. TouchTek4 
touchscreens are engineered to accept more than three million touches to any area of 
the screen. 

Barcode Reader 

The barcode reader (103), connected to user workstation (104) is used to 
25 identify the documents and pages of the documents selected by the user. One example 
of barcode reader it is possible to use, is the PenWAND © ComputerWare. The 
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PenWAND is a combination of barcode reader and light pen in one liand-held 
instrument. Tliis is an ideal solution for applications requiring reading of barcodes and 
screen selection. 

User Workstation 

5 In tlie embodiment of the invention illustrated in Figure 1 , the user workstation 

(104) is used for carrying out the following steps : 

• receiving from the barcode reader (103) a code for identifying (107) the document 
(101) (for instance, an address or an URL Uniform Resource Locator) and the page 
number (301) selected by the user; 

10 • receiving from the touch foil (102), (602) the coordinates of the two points (605), 
(606) touched by the user over the document (601), said points (the start point and 
the end point) specifying the part of the selected document page that must be read; 

• transmitting to a given server (106) means for identifying on said server an 
15 electronic copy (or replica, or representation) (900) of the physical document (101), 

(700) and data related to the part of the physical document (701), (702) selected by 
the user; said identification means comprising : 

• the document identification code (107), 

• the page identification code (301) and 

20 • the coordinates of the two points (605), (606) specifying the part (701), (702) of 

the physical document (700) selected by the user; 

• retrieving from the electronic copy of the document on the server (106), data (903) 
corresponding to the part (701 , 702) of the document (700) selected by the user; 

• extracting text data from said data; 

25 • converting this text data to speech (1 1 05) by means of a text-to-speech software; 

• optionally, magnifying on the user's display, the text data or the electronic copy 
corresponding to the selected part (1106) by means of a display magnifying 
software. 
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The user workstation (104) is, for example, an Internet appliance, a multimedia 
Personal Computer (PC), a game console, a cell phone ... The touch foil (102) 
communicates with the user workstation by means of a cable, a wire pair, an infrared 
link, or a wireless link. In a particular embodiment, the user workstation (104) is 
5 connected to a communication network (105), preferably the Internet network and 
comprises a Web Browser application. 

Server 

The server (106) is primarily the repository where electronic representations 
(900) of physical documents (700) are stored. Each electronic representation is 
10 associated with a physical document (or publication) by means of the barcode printed 
on this physical document. The electronic representation of a document includes an 
exact copy or replica of the original document. This copy includes, for instance, 
scanned images of pages. 

The electronic representation may also include text, text descriptions of graphical 
15 elements (such as charts, graphs or illustrations). The server also provides users with 
additional information such as : 

• a speech description (205) of the selected document (such as title, authors, general 
information); and 

• speech instructions related to the selected document (206) (e.g., for informing the 
20 user about the position of the barcodes on pages of selected physical document or 

publication). 

Method for selecting a part of a physical documents and for providing an 
electronic reading and screen magnification of this selected part 

As shown in Figure 12, the present invention discloses a system, method and 
25 computer program for : 
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• selecting a part of a physical document (preferably a printed publication, like the 
document shown in Figure 1 entitled "Early British Kingdoms'), simply by 
touching with a fingertip (or pen) over this physical document; 

• identifying in an electronic copy (or replica) of the physical document, the part that 
has been selected by the user; 

• extracting text data comprised in the selected part, and 

• converting extracted text into electronic speech. 

The method comprises the steps of: 

• (1 201 ) identifying a physical document, said physical document comprising one or a 
plurality of pages; 

• (1202) selecting and identifying a page of said physical document, said page 
comprising one or a plurality of printed items; 

• determining (1204) the position of a first point pressed (e.g. with a fingertip or pen) 
on a touch foil placed and aligned (1203) over or under the identified first page, said 
first point corresponding to the start point of a part selected in said identified 
document by the user; 

if start point and end point of the selected part are on the same page : 

• determining (1204) the position of a second point pressed (e.g. with a fingertip or 
pen) on the touch foil placed and aligned (1203) over or under the identified page of 
said document, said second point corresponding to the end point of said selected 
part; 

if start point and end point are not on the same page : 

• identifying a second page selected by the user in said physical document, preferably 
by reading, by means of a barcode reader, a page identifier printed on said second 
page at a predefined position; 

• determining (1204) the position of a second point pressed (e.g. with a fingertip or 
pen) on the touch foil placed and aligned (1203) over or under the identified second 
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page of said document, said second point corresponding to tlie end point of said 
selected part; 

• (1205) sending to a given server a request for the selected document part, said 
request comprising an identification of : 

5 • the physical document, 

• the page(s), and 

• the selected part of the document; 

• (1206) locating on the server an electronic copy (or replica) of the selected physical 
document; 

10 • (1207) extracting text data related to the part selected on the physical document; 

• (1208) receiving on the user workstation, the requested text data; 

• (1209) reading to the user, the received text data using a text-to-speech software on 
the user workstation. 



Figure 2 shows how the user selects a physical document (or publication) (201) 
15 (e.g., printed book entitled "Early British Kingdoms'), By means of any user interface 
(keyboard, mouse, voice recognition software and microphone) or any other reading 
means (e.g., barcode reader), the user enters a code or name (207) for identifying the 
document. In the preferred embodiment shown in Figure 2, the user reads by means of 
a barcode reader (203) a bar code (207) with the address on the Web (e.g.: the URL 
20 http://www.virgin.net/170301/ ) of the electronic copy (or replica with the same 
identical format) of the selected physical document (201). The bar code (207), used to 
identify the document, must be printed or attached at a predefined reserved place on 
the physical document (e.g., on the front cover, back cover or first page). 

The bar code gives access, on the sen/er (106), to information related to the 
25 selected physical document, such as: 

• the title (e.g., "Eariy British Kingdoms"), 

• the author's name (e.g., "David Nash Ford of Binfieid, Berl(shire, UK"), 

• the date of the publication (e.g., "28/01/ 1999") , 
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• the ISBN (International Standard Book Number) (e.g., "84-344-0856-2" ), etc. 

This information about the selected physical document is transmitted from the 
server (106) to the user workstation (204) as a text file. This text file is then converted to 
electronic voice (205) by means of a text-to-speech software operating on the user 
5 workstation. 

The user workstation can also receive from the server (106) operational 
instructions (206) for locating page numbers barcodes on the pages of the selected 
physical document or publication (e.g., "Page number codes are located on bottom 
left and bottom right corners of pages'). These operational instructions are 
10 transmitted to the user by means of the text-to-speech software operating on the user 
workstation. 

Figure 3 shows how a user opens a physical document (300) at the page where 
a bookmark has been inserted (e.g., on "Page 16"), This figure also shows how on each 
page of the document (e.g., "Page 16", "Page 17" ), a bar code (301), (302) with a 
15 page number, is printed on a predefined and fixed position, already known by the user. 

Figure 4 shows how a user selects a page of physical document (401) and how 
the number of the selected page is identified by scanning by means of barcode reader 
(403), the bar code (405) encoding the page number. After decoding the scanned bar 
code, the user workstation (404) informs the user (406) by means of an electronic 
20 voice, of the selected page number (e.g., "Page sixteen I"). 

Alternate embodiments for identifying pages of a document exists and can be 
used with the present invention. An example is given by a document having page 
numbers printed as text. It is possible to identify a page of the document by scanning 
on the page the place where the page number is printed. A scanning device can be 
25 integrated with the barcode scanner described previously. The scanning device can 
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also be used separately like the Quicklink Pen Personal Scanner described in the 
specifications of the present application. 

Figure 5 shows how a user places a touch foil (502) over a page of a physical 
document, aligning the borders of the touch foil and the document using some 
5 conventional means (505) (e.g., by adjusting the upper left corner of the touch foil with 
the upper left corner of the page). 

Figure 6 shows how, according to a particular embodiment of the present 
invention, it is possible to define a part of a physical document (601). The user presses 
(e.g., with a fingertip or pen) (1) on a first point (605) and then (2) on a second point 

10 (606) over the touch foil (602) placed aligned over the selected page (e.g., over "Page 
76" of the publication entitled "Early British Kingdoms'). When the user has finished 
to press the points defining the selected part that he desires to hear and that must be 
electronically read, she/he (3) then presses on the keyboard of the user workstation 
(604) a reserved READ key (607), informing the system to initiate the task of locating 

15 and reading the selected document part. Coordinates of the points (605), (606) pressed 
by the user on the touch foil (602) are transmitted from the touch foil to the user 
workstation (604). 

The selection of a part of a physical document using a touch foil, can be done 
according to various forms. For example, in a particular embodiment of the invention, if 

20 the user touches a single point on the touch foil and then presses the READ key, this 
operation is equivalent to a command for reading the full document from this point 
(possibly until the user presses a STOP READ reserved key on the user workstation 
keyboard). Also, if the user selects a first page and indicates a first point on this first 
page, and then selects a second page and indicates a second point on this second 

25 page, and then presses the READ key, this operation is equivalent to a command for 
reading the document from the first point selected on the first page to the second point 
selected on the second page. It is now apparent that other arrangements can be 
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implemented by those skilled in the art for correlating a plurality of points selected by a 
user with a part of a physical document. 

Figure 7 shows in detail two points (701), (702) on a page of a physical 
document (700). These two points correspond to the points (605), (606) pressed by the 
user on the touch foil (602). 

Figure 8 shows how the user workstation (804) sends a request (807) related to 
a part (701), (702) of a physical document (700) to a server (806). The request 
comprises an identification of: 

• the physical document, 

• the selected page, and 

• the coordinates of the part selected by the user. 

In the example illustrated in Figure 8, the HTTP request: 
http://www.vlrgin.net/1 70301/?&Page=16&Xf=27& Yf=43&Xt=152& Yt=168 

specifies: 

• the URL of the physical document (e.g., http://www.virgin.net/170301/), 

• the page number (e.g., "Page=16") , and 

• the coordinates of the part selected by the user (e.g., From: Xfs27, Yf=43 ; To: 
Xt=152, Yt=168). 

In the example illustrated in Figure 8, when the sen/er (806) (e.g., 
"www.virgin.net") receives the HTTP request, this request is parsed to find the 
physical document reference number (e.g., "770307" ). This physical document 
reference number (or code) is then used to search on the server for the electronic copy 
(e.g., "170301.PDF") of the physical document (same format and same content). 
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The HTTP request is also parsed to find the page of the physical document (e.g., 
"Page=16") selected by the user and the coordinates specifying the selected part of 
the physical document (e.g., From: Xfc27, Yf=43 ; To: Xt=152, Yt=168) on the sen/er. 

Figure 9 illustrate how, in a particular embodiment of the invention, the sen/er 
(806) uses the information comprised in the HTTP request (807) : 

• document reference number, 

• page number, and 

• coordinates of points defining the part of the document selected by the user, 

to locate and extract from the electronic copy (e.g., "170301.PDF" ) replica of the 
physical document (801), text data (903) (in bold) corresponding to the part of physical 
document (701), (702) selected by the user. The extracted text data is formatted by the 
sen/er in the form of a text file (e.g., "SELECTION.TXT') and is transmitted as reply to 
the request (807) previously received from user workstation (804). 

It is important to note that different methods can be used to determine, from the 
coordinates of the points selected (701), (702) on the touch foil, the text data (903) that 
must be must be extracted from the electronic copy (from point 901 to point 902). In the 
example illustrated in Figure 9, incomplete words are excluded from the head and from 
the tail of the text string comprised in the selected part of the physical document. 

The server can also receive from the user workstation a HTTP request (such as 
"&format=magnifyOriginar') to view a magnified image of a part of the original 
document. The server sends back a reply including an image of the selected part of the 
same format and appearance than the original document. This image is magnified on 
the display of the user workstation.. 

The information returned by the sen/er (806) in reply to the request (807) 
received from user workstation (804) can be different than the exact transcription (as a 
text file) of the data printed on the physical document. In fact, nothing prevents the 
server (806), to reply with any kind of multimedia information (e.g., a combination of a 
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long text description with music or songs). This multimedia infonnation can be linked (or 
even hyperlinked) with different parts of the physical document that the user can select. 
Therefore, the invention is not limited to a mechanism for reading (i.e., for speaking by 
means of text-to-speech transcription) text printed on a physical document to a blind 
5 user, but the invention is also directed to a mechanism for accessing from a physical 
document, all kinds of multimedia information or services available on the Web 

Figure 10 shows how, in reply to request of a user for accessing information 
from a selected part (701), (702) of a physical document, text data (903) extracted from 
(or associated with) the selected part on the electronic copy of the document (900), Is 
10 downloaded (1007) as a text file (e.g., "SELECTION.TXT") from the server (1006) to 
the user workstation (1004). 

Figure 11 shows how the text file (e.g., "SELECTION.TXT"), once received by 
the user workstation (1104), is converted into an audio file (.aiff, .wav, .au file) by 
means of a text-to-speech software and is played (1 105) to the user. This figure shows 
15 also, how the received text is magnified (1106) on the user workstation display by 
means of a screen magnifying software. 

What has been described is merely illustrative of the application of the principles 
of the present invention. Other arrangements and methods can be implemented by 
those skilled in the art without departing from the spirit and scope of the present 
20 invention. 
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